Is it possible to anticipate which patents utilizing diffusion models might be filed in the future? This column, the last installment of 'The Diffusion Model from a Patent Perspective,' aims to examine the latest research trends in technologies related to diffusion models and explore the outlook for future patents related to diffusion models.
The Research Trends in Diffusion Model-Related Technologies
As previously explained in the patent related to the diffusion model in the video domain, semantic information that can well represent the characteristics of the data is important in order to produce good results using a generative model. Through semantic information, the intent, concept, and interrelationships of the data can be known, and the generative model can use this to produce good results. For example, an image may have the relationship between pixels as semantic information, and a video may contain the time series information of each frame and the relationship between frames as semantic information in addition to the relationship between pixels. The video domain can be considered to have a larger amount of important information compared to the image domain.
Therefore, when generating results in the video domain through diffusion models, in addition to considering the relationships between pixels, it is necessary to take into account the relationships between individual frames. This ensures that static areas are represented consistently, while changes are introduced only in areas with motion. Consequently, learning in the video domain is more challenging compared to the image domain, and enhancing the quality of the results becomes a complex task.
This issue is not unique to the video domain but is a common challenge found in data domains that hold a significant amount of important information, such as 3D images, audio data, and other data domains, in comparison to 2D images.
<Imagen Video (Source: Youtube, Google’s Video AI: Outrageously Good! 🤖)>
One way to address these issues has been through various research efforts. For instance, Google, which had garnered significant attention for its remarkable results using diffusion models in the image domain, introduced Imagen Video (link) in 2022, a technology that can generate videos from text.
<Top 10 Papers by CVPR Category>
Furthermore, when examining the papers submitted to 'CVPR 2023,' the recent conference in the field of computer vision and pattern recognition, it is noted that the area with the highest number of submitted papers is the "3D image generation domain," followed by the "video generation research domain." Considering this situation, one can infer that research on diffusion models that can be applied to data domains beyond images is actively underway.
On the other hand, within various research endeavors aimed at utilizing diffusion models, the persistent issues of diffusion models, such as generation speed and lightweighting, have been somewhat addressed through Stable Diffusion. Moreover, with many easily accessible large language models like OpenAI's Chat GPT (GPT-4) and Google's Bard starting to support multi-modal capabilities, various possibilities are emerging in the application directions of diffusion models.
Regarding patents, unless registered or applicants request early publication, "Published Patent Application" is typically published "1 year and 6 months from the filing date." Consequently, it can be challenging to immediately assess the status of patents related to the latest technology as soon as they are filed. Nevertheless, given the active research in academia in data domains beyond images, it can be surmised that many patents applying diffusion models to various domains have already been filed but are yet to be publicly disclosed.
<Why is this image peculiar? (Source: GPT-4 Technical report)>
When considering recent research trends related to patents and diffusion models as introduced earlier, it is expected that many inventions utilizing diffusion models in various modalities or inventions that combine diffusion models with other models will emerge. Therefore, when commercializing technologies related to diffusion models, it will be crucial to anticipate research trends related to the applications of diffusion models through patents to secure rights.
Through three columns so far, we have discussed the significance of diffusion models, quantitative and qualitative analyses of the status of patent applications related to diffusion models, and the future outlook of patents related to diffusion models.
As observed, the number of patent applications related to diffusion models has dramatically increased since 2021, and the content of patents is expanding from simple image domains to other domains like video and audio. Moreover, considering the recent research trends, the increasing trend of patent applications related to diffusion models is expected to continue for some time.
In cases where technology is publicly disclosed through papers and similar means but is not followed by subsequent patent applications, exclusive rights cannot be claimed for extensively researched technology. This is especially critical in highly competitive fields of technology like diffusion models, where it is essential to proactively protect the results of research.
